Reliability and Agreement of Ultrasonographic Measures of the Ovarian Stroma: Impact of Methodology.

OBJECTIVES: Increased ovarian stromal area (SA), stromal-to-ovarian area ratio (S/A), and echogenicity (SEcho) on ultrasonography have been proposed as diagnostic markers for polycystic ovary syndrome. Although several methods to evaluate the stroma exist, their reproducibility has not been defined which limits clinical utility. This study aimed to determine the interrater reliability and agreement of methods to evaluate SA, S/A, and SEcho. METHODS: Five raters tested 3 methods to obtain SA and S/A, and one to obtain SEcho on 30 ovarian cineloops under two imaging conditions, simulating real-time (free-choice) or offline (fixed-frame) imaging. For SA, Method 1 subtracted follicular area from the ovarian area, Method 2 involved outlining the periphery of the stroma, and Method 3 represented a hybrid approach in which central follicles were subtracted from the outlined stroma. SEcho was scored on a subjective 3-tiered scale. Intraclass correlation coefficients (ICCs) and the coefficient of variation were determined for SA and S/A, and Fleiss' kappa agreement statistics (κ) were determined for SEcho. RESULTS: Interrater reliability of SA was superior using Method 1 (ICC = 0.558 and ICC = 0.705) versus Method 2 (ICC = 0.522 and ICC = 0.230) or Method 3 (ICC = 0.429 and ICC = 0.305) under free-choice and fixed-frame imaging conditions, respectively. Interrater reliability of S/A was also moderate to poor across methods. SEcho was also not reliably assessed across raters (κ = <0.500). CONCLUSIONS: Ultrasonographic assessments of the ovarian stroma were associated with moderate to poor reproducibility. Indirect estimates of the ovarian stroma (Method 1) could be optimized to yield a reproducible approach, clarifying the clinical relevance of the stroma.

Sonographic markers of ovarian morphology, but not hirsutism indices, predict serum total testosterone in women with regular menstrual cycles.

OBJECTIVE: To determine whether sonographic markers of ovarian morphology or male pattern hair growth scores predict androgen levels in women with regular or irregular menstrual cycles. DESIGN: Cross-sectional observational study. SETTING: Clinical research unit. PATIENT(S): Seventy-six women of reproductive age (18-39 years) were evaluated for male-pattern hair growth (using a modified Ferriman-Gallwey scoring system), ovarian morphology (by transvaginal ultrasonography), and total serum testosterone (T) (by liquid chromatography tandem mass spectrometry). INTERVENTION(S): Not applicable. MAIN OUTCOME MEASURE(S): Regional and total modified Ferriman-Gallwey scores, number of follicles per follicle size category, follicle number per ovary, ovarian volume, ovarian area, stromal to ovarian area ratio, stromal echogenicity index, total testosterone (total T), and menstrual cycle length. RESULT(S): Neither regional nor total modified Ferriman-Gallwey scores correlated with total T concentrations in women with regular or irregular menstrual cycles, as judged by the Least Absolute Shrinkage and Selection Operator technique. By contrast, a sonographic marker (follicle number per ovary 6-9 mm) significantly predicted total T concentrations in women with regular menstrual cycles but not in women with irregular menstrual cycles. CONCLUSION(S): Sonographic markers of ovarian morphology, but not hirsutism scores, predicted total T levels. However, the predictive value of ovarian morphology for total T differed by menstrual cycle status. That sonographic markers did not predict androgen levels in a diverse cohort of women with cycle irregularity suggests the potential for distinct variations in ovarian morphology for androgenic and nonandrogenic types of cycle irregularity. Overall, our findings support that an assessment of ovarian morphology may be helpful in reflecting total T levels.

Updated ultrasound criteria for polycystic ovary syndrome: reliable thresholds for elevated follicle population and ovarian volume.

STUDY QUESTION: Do the ultrasonographic criteria for polycystic ovaries supported by the 2003 Rotterdam consensus adequately discriminate between the normal and polycystic ovary syndrome (PCOS) condition in light of recent advancements in imaging technology and reliable methods for estimating follicle populations in PCOS? STUDY ANSWER: Using newer ultrasound technology and a reliable grid system approach to count follicles, we concluded that a substantially higher threshold of follicle counts throughout the entire ovary (FNPO)-26 versus 12 follicles-is required to distinguish among women with PCOS and healthy women from the general population. WHAT IS KNOWN ALREADY: The Rotterdam consensus defined the polycystic ovary as having 12 or more follicles, measuring between 2 and 9 mm (FNPO), and/or an ovarian volume (OV) >10 cm(3). Since their initial proposal in 2003, a heightened prevalence of polycystic ovaries has been described in healthy women with regular menstrual cycles, which has questioned the accuracy of these criteria and marginalized the specificity of polycystic ovaries as a diagnostic criterion for PCOS. STUDY DESIGN, SIZE, DURATION: A diagnostic test study was performed using cross-sectional data, collected from 2006 to 2011, from 168 women prospectively evaluated by transvaginal ultrasonography. Receiver operating characteristic (ROC) curve analyses were performed to determine the appropriate diagnostic thresholds for: (i) FNPO, (ii) follicle counts in a single cross section (FNPS) and (iii) OV. The levels of intra- and inter-observer reliability when five observers used the proposed criteria on 100 ultrasound cases were also determined. PARTICIPANTS/MATERIALS, SETTING, METHODS: Ninety-eight women diagnosed with PCOS by the National Institutes of Health criteria as having both oligo-amenorrhea and hyperandrogenism and 70 healthy female volunteers recruited from the general population. Participants were evaluated by transvaginal ultrasonography at the Royal University Hospital within the Department of Obstetrics, Gynecology and Reproductive Sciences, University of Saskatchewan (Saskatoon, SK, Canada) and in the Division of Nutritional Sciences' Human Metabolic Research Unit, Cornell University (Ithaca, NY, USA). MAIN RESULTS: Diagnostic potential for PCOS was highest for FNPO (0.969), followed by FNPS (0.880) and OV (0.873) as judged by the area under the ROC curve. An FNPO threshold of 26 follicles had the best compromise between sensitivity (85%) and specificity (94%) when discriminating between controls and PCOS. Similarly, an FNPS threshold of nine follicles had a 69% sensitivity and 90% specificity, and an OV of 10 cm(3) had a 81% sensitivity and 84% specificity. Levels of intra-observer reliability were 0.81, 0.80 and 0.86 when assessing FNPO, FNPS and OV, respectively. Inter-observer reliability was 0.71, 0.72 and 0.82, respectively. LIMITATIONS, REASONS FOR CAUTION: Thresholds proposed by this study should be limited to use in women aged between 18 and 35 years. WIDER IMPLICATIONS OF THE FINDINGS: Polycystic ovarian morphology has excellent diagnostic potential for detecting PCOS. FNPO have better diagnostic potential and yield greater diagnostic confidence compared with assessments of FNPS or OV. Whenever possible, images throughout the entire ovary should be collected for the ultrasonographic evaluation of PCOS. STUDY FUNDING AND COMPETING INTEREST: This study was funded by Cornell University and fellowship awards from the Saskatchewan Health Research Foundation and Canadian Institutes of Health Research. The authors have no conflict of interests to disclose.

Grid analysis improves reliability in follicle counts made by ultrasonography in women with polycystic ovary syndrome.

Poor reliability has been reported when counting the total number of follicles in polycystic ovaries using conventional two-dimensional (2-D) ultrasound viewing methods. In the current study, we report good reliability in follicle counts when observers imposed a programmable grid system over the viewing window. Four observers estimated total follicle counts in 45 ovarian ultrasound scans by compartmentalizing the ovary into 9 to 12 grid sections and performing focused follicle counts per section. The mean number of follicles counted per ovary was 44.6 +/- 2.3. The level of inter-observer agreement when making follicle counts was 0.82 and total follicle counts did not differ among observers. The level of intra-observer agreement was 0.93 which further corroborated the utility of this method for making dependable follicle counts. In summary, the ability to obtain reproducible follicle counts will help to establish reliable diagnostic criteria for polycystic ovarian morphology.

Assessment of ultrasonographic features of polycystic ovaries is associated with modest levels of inter-observer agreement.

BACKGROUND: There is growing acceptance that polycystic ovaries are an important marker of polycystic ovary syndrome (PCOS) despite significant variability when making the ultrasound diagnosis. To better understand the nature of this variability, we proposed to evaluate the level of inter-observer agreement when identifying and quantifying individual ultrasonographic features of polycystic ovaries. METHODS: Digital recordings of transvaginal ultrasound scans performed in thirty women with PCOS were assessed by four observers with training in Radiology or Reproductive Endocrinology. Observers evaluated the scans for: 1) number of follicles >/= 2 mm per ovary, 2) largest follicle diameter, 3) ovarian volume, 4) follicle distribution pattern and 5) presence of a corpus luteum (CL). Lin's concordance correlation coefficients and kappa statistics for multiple raters were used to assess inter-observer agreement. RESULTS: Agreement between observers ranged from 0.08 to 0.63 for follicle counts, 0.27 to 0.88 for largest follicle diameter, 0.63 to 0.86 for ovarian volume, 0.51 to 0.76 for follicle distribution pattern and 0.76 to 0.90 for presence of a CL. Overall, reproductive endocrinologists demonstrated better agreement when evaluating ultrasonographic features of polycystic ovaries compared to radiologists (0.71 versus 0.53; p = 0.04). CONCLUSION: Inter-observer agreement for assessing ultrasonographic features of polycystic ovaries was moderate to poor. These findings support the need for standardized training modules to characterize polycystic ovarian morphology on ultrasonography.

Improving inter-observer variability in the evaluation of ultrasonographic features of polycystic ovaries.

BACKGROUND: We recently reported poor inter-observer agreement in identifying and quantifying individual ultrasonographic features of polycystic ovaries. Our objective was to determine the effect of a training workshop on reducing inter-observer variation in the ultrasonographic evaluation of polycystic ovaries. METHODS: Transvaginal ultrasound recordings from thirty women with polycystic ovary syndrome (PCOS) were evaluated by three radiologists and three reproductive endocrinologists both before and after an ultrasound workshop. The following endpoints were assessed: 1) follicle number per ovary (FNPO), 2) follicle number per single cross-section (FNPS), 3) largest follicle diameter, 4) ovarian volume, 5) follicle distribution pattern and 6) presence of a corpus luteum (CL). Lin's concordance correlation coefficients (rho) and kappa statistics for multiple raters (kappa) were used to assess level of inter-observer agreement (>0.80 good, 0.60 - 0.80 moderate/fair, <0.60 poor). RESULTS: Following the workshop, inter-observer agreement improved for the evaluation of FNPS (rho = 0.70, delta rho = +0.11), largest follicle diameter (rho = 0.77, delta rho = +0.10), ovarian volume (rho = 0.84, delta rho = +0.12), follicle distribution pattern (kappa = 0.80, delta kappa = +0.21) and presence of a CL (kappa = 0.87, delta kappa = +0.05). No improvement was evident for FNPO (rho = 0.54, delta rho = -0.01). Both radiologists and reproductive endocrinologists demonstrated improvement in scores (p < 0.001). CONCLUSION: Reliability in evaluating ultrasonographic features of polycystic ovaries can be significantly improved following participation in a training workshop. If ultrasonographic evidence of polycystic ovaries is to be used as an objective measure in the diagnosis of PCOS, then standardized training modules should be implemented to unify the approach to evaluating polycystic ovarian morphology.